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Abstract 


Higher education (HE) today faces high requirements such as digitalisation and challenges such as 
internationalisation, as a response to globalisation. Virtual Mobility (VM) has a great potential to 
contribute to both digitalisation and internationalisation of HE. Mobility of students and staff has been 
one of the central objectives and main policy areas of the European Higher Education Area (EHEA), 
since it enhances development of competencies, increases cooperation and competition, fosters 
respect for diversity, and encourages linguistic pluralism underpinning the multilingual tradition of 
Europe. In recent years, the options for extension of virtual mobility have increased through the 
advent of Massive Open Online Courses, Open Educational Resources, Open Credentials and Open 
Educational Policies. The paper introduces a framework for Quality Assurance (QA) for attainment, 
assessment and recognition of Virtual Mobility skills in view of Open Education in the European 
context. OpenVM is a strategic partnership for innovation and the exchange of good practices founded 
under Erasmus+ program and is composed of nine European partner organisations from HE. The 
project aims at enhancing the uptake of VM in HE by improving VM skills of HE teachers and students 
though Open Education practices. The paper presents the components of the QA framework and 
provides insights into QA design and evaluation, including empirical data from on online survey 
assessing fitness-for-purpose of the OpenVM QA framework. 
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1 INTRODUCTION 


Higher education (HE) today faces high requirements such as digitalisation and challenges such as 
internationalisation, as a response to globalisation. Virtual Mobility (VM) has a great potential to 
contribute to both digitalisation and internationalisation of HE. Mobility of students and staff has been 
one of the central objectives and main policy areas of the European Higher Education Area (EHEA), 
since it enhances development of competencies, increases cooperation and competition, fosters 
respect for diversity, and encourages linguistic pluralism underpinning the multilingual tradition of 
Europe. In recent years, the options for extension of virtual mobility have increased through the advent 
of Massive Open Online Courses, Open Educational Resources, Open Credentials and Open 
Educational Policies. In this context, Quality Assurance (QA) has been gaining importance especially 
in view of setting common standards and evaluation criteria, such as the ones related to assessment 
and recognition of learning. The common challenge has been to develop new forms of collaboration 
across institutions and nations and at the same time take into account diversity and plurality of higher 
education [1]. 


This paper presents a QA framework of the European project Open Virtual Mobility (OpenVM) 
including preliminary evaluation results. OpenVM explores synergies between Open Education (OE) 
and Virtual Mobility (VM). The project addresses some of the challenges related to adoption of VM in 
HE in Europe at a wider scale and aims to create a European Learning Hub for achievement, 
assessment and recognition of VM skills. The VM Learning Hub is envisaged to become a central 
reference point for educators and students wishing to learn about the different possibilities and forms 
of virtual mobility, find and learn from good practice examples, assess and recognise own VM skills, i. 
e. skills acquired from and/or relevant for the implementation and/or participation in VM actions [2]. 
The project has been developing a set of tools and guidelines to enhance achievement, assessment 
and recognition of VM skills, provide support for the design and implementation of VM, and enhance 
collaborations of HE educators and students. The VM Learning Hub aims to provide engaging learning 
experiences and produce evidence about how achievement, assessment and credentialing of VM 
skills contributes to the uptake virtual mobility [2]. 
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The QA framework developed in OpenVM project is crucial for ensuring quality-oriented 
implementation of the project. It guides the work inside and outside the partnership including 
stakeholders inside HE (educators, students, leaders, international offices) and outside HE (e. g. 
policy-makers). The QA approach in OpenVM applies concepts from innovation theory, such as the 
“community of innovation” approach [3], for defining the QA framework. 


2 FRAMEWORK 


A fundamental aspect for ensuring the success of the OpenVM project is the QA framework. Conole 
[4, p. 3] cites the definition of quality as “the standard of something as measured against other things 
of a similar kind; the degree of excellence of something: quality of life’. The QA framework has been 
described as “the means through which an institution ensures and confirms that the conditions are in 
place for students to achieve the standards set by it or by another awarding body” [4, p.3]. 


The QA framework in OpenVM project is rooted in the Design Based Research model (DBR) [5], 
conceptualised as a cyclic implementation process in the creation of a didactic product (in its wider 
definition). This cyclic conceptualisation emerges from the application of pedagogical models which, 
based on instructional design, come together into the model of design based research [5]. This model 
suggests the adoption of a process including the following stages, known by its acronym ADDIE: 
Analysis, Design, Development, Implementation, Evaluation. McKenney & Visscher-Voerman [6] build 
on the model by Mckenney & Reeves [7] and propose a synthesis of existing DBR approaches. 


Following Salinas [8], this methodological approach has achieved a substantial uptake in the research 
field of technology-enhanced education, since it aspires to build knowledge on the design, 
development and assessment of the educational process. This research paradigm emerges from the 
failure of existing ones to impact and improve educational practices in formal settings, and its main 
aim is to explore specific and contextualised problems that need to be solved [9]. Furthermore, DBR is 
suitable to study innovation processes. In this case, it creates innovative educational contexts by 
contrasting theory in practice, observe action and create knowledge in the successive iterations of 
innovation [9]. 


The QA in the OpenVM project is based on several existing QA frameworks and guidelines from which 
the diverse QA approaches are derived. These are: (a) ENQA recommendations for HE institutions 
[10] and the updated version ESG by ENQA and other agencies [11], (b) Updated Quality Assessment 
for E-learning [12], (c) OpenEdu Framework [13], (d) Virtual Mobility Matrix by EADTU [14], and (e) 
guidelines by Jansen, Rosewell and Kear [15], which are used for QA of the OpenVM MOOC. 


2.1 Internal and External Quality Assurance 


Following ENQA and ESG recommendations for HE institutions, the OpenVM QA framework 
encompasses standards for internal and external QA with specific procedures and guidelines for 
internal and external QA [10, pp. 16-23], [11, pp. 11-20]. Some standards are applied to the project as 
whole, while other standards are applied to particular parts of the project, such as the OpenVM 
MOOC. Internal QA in the project is based on the following guidelines: 


1 Policy and procedures for quality assurance: The project is committed to a culture of quality and 
has dedicated resources to develop the QA framework for the implementation of the entire 
project and its components, such as quality assurance for VM skills development, assessment 
and recognition. The QA guidelines specify the organisation of the internal and external quality 
assurance system. 


2 Monitoring and periodic review: The DBR model in which the project is rooted specifically helps 
to guarantee the permanent improvement of the program. The project has mechanisms such as 
the QA framework itself to ensure quality and conduct regular periodic reviews inside the 
partnership (such as bi-monthly survey) and with external QA experts (expert evaluations), both 
providing regular feedback on quality assurance and project activities. Internal and external 
reviews have been also integrated into the QA framework for the production of outputs. 


3 Learner-centred attainment, assessment and recognition of skills: The OpenVM project is 
dedicated to learner-centered attainment, assessment and recognition of VM skills. OpenVM 
MOOC is designed to address the diversity of learners (students, teachers, international 
officers) and the need to promote open and self-regulated learning. OpenVM e-assessment and 
digital credentials are designed in alignment with existing competency-frameworks. The criteria 
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for assessment and recognition of skills with digital credentials are co-designed with project 
partners and external experts to be transparent and available for verification checks. 


4 Learning resources and learner support: The OpenVM Learning Hub is designed to include 
OER and MOOC to assist learners in attainment, assessment and recognition of their VM skills. 
The Learning Hub applies principles of meaningful gamification with focus on creating 
personalisable and customisable learning experiences. The designed is developed and tested 
with relevant stakeholder groups. 


5 Information systems and management: the quality-related information system in the OpenVM 
MOOC is expected to cover: profile of the learner population; learner progression and success 
rates; learner satisfaction with OER and MOOC; diverse key performance indicators, such as 
number of visits to the VM Learning Hub. 


6 Public information: OpenVM regularly publishes up to date, impartial and objective information 
on the project website: http:/Avww.openvirtualmobility.eu/ including both quantitative and 
qualitative information, about the project and its activities. 


In relation to specific outputs of the projects, such as the design and implementation of the OpenVM 
MOOC, the QA framework utilised principles from specific frameworks such as the QA framework for 
e-learning [12]. This framework supports the decision-making process in diverse areas such as 
strategic policy, curriculum and course design, definition of skills and assessment criteria as well as 
course design and learner support. 


External QA in the project is based on the following guidelines based on [11, p. 19 ff.]: 
1 Quality: External QA addresses the effectiveness of the internal QA 


2 Implementation: External QA is designed to ensure its fitness for purpose to achieve the aims 
and objectives of the project. Stakeholders are involved in the design and improvement of the 
QA. This includes evaluations by experts, educators and students based on the review model 
with diverse steps from self-evaluation to the published QA report. 


3 Criteria. External QA is based on explicit and published criteria which are applied consistently, 
irrespective of whether the process leads to a formal decision. QA is designed to be reliable, 
useful, pre-defined, implemented consistently and published. 


4 Reporting. External QA includes reports by the experts to be published and made accessible to 
the academic community, external partners and other stakeholders. 


5 Complaints. External QA includes procedures for handling complaints, e. g. the OpenVM 
MOOC will be designed to enable participants to raise issues of concern. 


2.2 Open Education and Quality Assurance 


Furthermore, the focus and context of the project leads to the need to include the principles of Open 
Education (OE) in the QA framework. OpenVM follows the principles of the OpenEdu Framework by 
the European Commission [13] to guarantee alignment with OE. 


The OpenEdu Framework [13] consists of ten dimensions, which describe “aspects or features of a 
situation” and “consist of different courses of action, focusing on a given area, which interact and 
together shape the practices of open education” [13, p. 24]. There are two dimensions: core and 
transversal. The core dimensions are the most common in Open Education Practices (OEP) and 
answer the question of “what” when opening up education. These include: access, content, pedagogy, 
recognition, collaboration and research. The transversal dimensions answer to the question of “how” 
of opening up education. These include: leadership, strategy, quality and technology. 


The core dimensions can be briefly summarised as follows [13, pp. 25-27]: 


1 Access. The VM Learning Hub provides open access based on its three levels: access to 
program, courses and educational content, e. g. OpbenVM MOOC, OER. 


2 Content. OER will be based on a wide range of formats, and will have different granularity, from 
the single object for a piece of content to the full course (MOOC). 


3 Pedagogy. The learning design will explore a wide variety of practice with digital tools, making 
the rationale available both for the learning and assessment design. 
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Recognition. The recognition of VM skills is to be conducted with open digital credentials which 
make skill descriptions and assessment criteria transparent. 


Collaboration. Collaboration is at the heart of VM Learning Hub together with the MOOC 
designed to promote collaboration among students, educators and and other stakeholders. This 
includes activities to co-create and exchange OER and other OEP. 


Research. The research linked to the design, implementation and assessment is carried out 
under the umbrella of openness, promoting collaboration and generating data for further 
common work. 


The transversal dimensions can be described in the following way [13, pp. 27- 29]: 


1 


Strategy. The whole partnership is carried out under the strategy of open practices, including 
the design of the VM Learning Hub, MOOC, OER and open credentials. 


Technology. All technology solutions are committed to OEP and are designed in relation to the 
ten core dimensions and other needs emerging from the interrelation of both core and 
transversal ones. 


Quality. In the context of open education it refers to the convergence of five concepts as 
highlighted by the JRC IPTS report [16]: efficacy (fitness for purpose of the content), impact 
(related to the extent to which an element is effective), availability (related to transparency and 
ease-of-access), accuracy (referring to the lack of mistakes) and excellence (in relation to other 
similar objects and its quality-potential). 


Leadership. The project is developed with the final objective of fostering Open Virtual Mobility 
across European Higher Education institutions. 


Based on the OpenEdu Framework [13] a number of strategic decisions have yet to be made in the 
project, e. g. (a) implementation of learning analytics to gather and assess data in VM Learning Hub, 
(b) MOOC design to maximise flexibility, modularity, academic community development, transferable 
skills, and assessment procedures (both formative and summative), and (c) support for users 
regarding technology and educational aspects and also considering the different roles (educators, 
students, international officers etc.). 


Also the MOOC design has to take into consideration the problem of the low completion rates in 
MOOCs in general and bear in mind some key quality criteria as proposed by [15]: 


a) 
b) 


quality from the learners’ point of view (rationale, backgrounds and abilities), 


quality connected to the pedagogical framework of the MOOC (autonomy, diversity, openness 
and interactivity), 


quality related to the input elements (instructional design, content and resources, technology 
and teachers), 


quality based on outcome measures, considering that low completion and certification are not 
necessarily due to poor quality and thus, more metrics and data are needed. 


3 METHODOLOGY 


A paper The design of the QA framework in the OpenVM project follows the three overall objectives: 


4 


Design of QA standards for the VM Learning Hub and its components (such as collaborative 
learning activities for OER co-design, algorithmic mechanism for group formation in the VM 
MOOC, e-assessment and recognition of VM skills with open credentials such as Open Badges 
and Blockcerts); 


Design of QA processes for linking and integrating quality standards to all stages of 
development of intellectual outputs, e. g. by applying effective design and development 
methodology such as quality-driven agile development, peer-reviews of iterations and final 
outcomes as well as quality implementation process and corrective actions. 


Design of QA instruments for quality assessment of teamwork (such as_ bi-monthly 
assessments) based on a set of qualitative and quantitative radiators (indicators) such as mood, 
agility, congruence and velocity. 
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Based on the previous international frameworks, diverse tasks and instruments have been included in 
the QA for the OpenVM project, which has been designed, implemented and assessed in an iterative 
cycle following the DBR model. The following table presents the three QA objectives with their tasks 
and instrument planning: 


Table 1. Quality Assurance framework in the OpenVM project: objectives and instruments. 


Objectives Instruments 
Design of QA QA standards are based on the following documents: 
Standards 


« | ENQA recommendations for HE institutions [10] 

* ESG Standards and Guidelines for QA in the EAHE [11] 
* Quality Assessment for E-learning [12] 

* OpenEdu Framework [13] 

¢ — Virtual Mobility Matrix by EADTU [14] 


* Guidelines by Jansen, Rosewell and Kear [15] 


Design of QA QA processes are based on DBR cycles for peer-reviews of 
Processes outcomes: 


« Assessment by partners (internal) 
« After improvement 2nd assessment by external experts 


« After improvement 3rd assessment by pilot users 


Design of QA QA instruments include: 


Instruments : ‘ : 
¢  Bi-monthly online partner survey (internal) 


« Evaluation by external experts 


« User testing (pilot users) 


To assess the QA framework including its methodology as described above, an online survey has 
been designed and answered by internal and external stakeholders. The results of the first evaluations 
are presented below. 


4 RESULTS 


The The evaluation results presented here are those of the first stage of the analysis, half-way through 
the process. Following the guide by European Commission [17] and the work by Escobar-Pérez and 
Cuervo-Martinez [18], the evaluation of the QA framework was based on three general criteria: 
relevance, appropriateness and feasibility. Relevance is the degree to which aims and needs are met. 
Appropriateness refers to whether an item is a substantial part of the conceptual framework. 
Feasibility describes the possibilities to be carried out. Thus, the reviewers assessed the extent to 
which the QA framework is relevant, appropriate and feasible for fulfilling the objectives of the project. 
The criteria were assessed on a scale from 1 to 5, i. e. from total disagreement (1) to total agreement 
(5). At the end of the survey there were open questions for reviewers to provide their insights and 
recommendations related to the coherence of the QA framework. Reviewers could also add any other 
document or instrument they consider necessary or useful. The results of the online survey assessing 
fitness-for-purpose of the openVM QA framework is presented below based on the participations of 
four internal and three external experts (n = 7) from Spain, Italy, Germany, France and Belgium. 


The evaluation by internal reviewers in general achieved a higher level of agreement about relevance, 
appropriateness and feasibility altogether with ratings mainly between 4 and 5 points on the scale. The 
evaluation by external reviewers shows a greater disagreement: although the OpenVM QA framework 
is considered relevant, it is considered less appropriate compared to internal partners, and also 
achieves the lowest score on feasibility. The key problem with feasibility reported by the reviewers is 
that is is time consuming, especially from the perspective of external reviewers. The figures below 
present selected results from the survey (cf. Figure 1 and 2). 
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Q1: 1st Cycle DBR Learning Hub Design 


relevance (5) 


appropriateness (5) 


feasibility (5) 


0 25 50 75 100 
Wi External reviewers @ Internal reviewers 


Figure 1. Evaluation by external and internal reviewers (n =7) of 1st DBR cycle for Learning Hub 
Design, scale 1 (low) to 5 (high), results are shown in percents for the 5-point score. 


Figure 1 indicates high scores of 5 on all three criteria — relevance, appropriateness and feasibility for 
the 1st DBR cycle which is internal assessment by partners related to the design of the VM Learning 
Hub. All reviewers seem to agree that the design of this QA cycle is relevant, appropriate and feasible 
for the project. The results for the evaluation of the 2nd DBR cycle related to the design of the VM 
Learning Hub show greater discrepancies — the majority of reviewers, i. e. 57% (4 out of 7), rated the 
relevance of the 2nd cycle with 4 points and only 2 out of 7 with 5 points, 43% (3 out of 7) rated 
appropriateness with 4 points and and only 2 out of 7 with 5 points, while feasibility was rated by 43% 
(3 out of 7) with 5 points with other scores for feasibility ranging from 1 to 3 points. The 3rd DBR cycle 
(i. e. testing with pilot users) shows again a more aligned scoring as in case of the 1st DBR cycle with 
57% assessing relevance with 5 pints and appropriateness with 4 points. The feasibility of the 3rd 
cycle is rated with 5 points by 86% (6 out of 7) reviewers. 


Figure 2 below indicates high scores for the assessment of the quality standards in the OpenVM QA 
framework applied to the 1st DBR cycle of the joint work on project outputs at all stages of the 
development. All external reviewers assessed the application of the quality standards in the 1st DBR 
cycle as appropriate and 66% (2 out of 3) also assessed the relevance and feasibility of quality 
standards with 5 points. The scores for the 2nd and 3rd cycle are very dispersed with individual rating 
ranging from 1 to 5 and scores for the 3rd phases slightly higher than for the 2nd phase. These 
selected results couples with results presented in figure 1 indicate expert agreement on relevance, 
appropriateness and feasibility of the 1st cycle, i. e. internal quality assessment by project partners 
and expert disagreement, especially related to the 2nd cycle, i. e. evaluation by external experts. 


Q2: 1st Cycle DBR Project Outputs 


relevance (5) 


appropriateness (5) 


feasibility (5) 


0 25 50 75 100 
Wi External reviewers ® Internal reviewers 


Figure 2. Evaluation by external and internal reviewers (n =7) of 1st DBR cycle for Learning Hub Design, 
scale 1 (low) to 5 (high), results are shown in percents for the 5-point score. 
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In general, the QA framework coherence is scored as relevant, appropriate and feasible, and some 
comments are made about the possibilities of getting an overview of the project work and outcomes. 
The reviewers have also left some critical yet helpful remarks in the survey, e. g. the implementation 
seems to be very time-consuming and create an excessive amount of work, especially to ask external 
experts who participate in the evaluation on a voluntary basis. Reviewers recommend further tools 
such as SWOT analysis and a scope planning which will be considered in the partnership to be added 
in the QA framework. 


5 CONCLUSIONS 


After a general overview of the QA framework and the results of the first evaluation with internal and 
external experts, first conclusions about the OpenVM QA framework can be already highlighted at this 
stage: Although diverse iterations in the DBR model are mandatory in order to carry out a completed 
cycle of innovation, the barriers of time-consuming work and assessment workload, especially of 
external experts participating in evaluations on a voluntary basis, may suggest the need for balancing 
efforts and reducing cycles of iteration if necessary. On the one hand, the whole DBR cycle seems 
very much needed for the design, implementation and assessment of the VM Learning Hub and its 
components such as OER, MOOC, open credentials. On the other hand, it seems that the internal QA 
processes need shorter review cycles and a more compact for external reviews. In order to improve 
the current QA framework it may be also useful to self-evaluate perceived notions of quality related to 
standards, processes and instruments within the partnership and based on the results set priorities for 
quality assurance for the next phases. Once the priorities are determined, the prioritised standards, 
processes and instruments need to be documented, e. g. in form of the QA manual. The QA manual 
can be articulated as reference standards with well-defined and clearly stated criteria and work-flows 
for internal and external QA, as well as and clear instructions to guide project partners in working 
together on project outcomes. Following the recommendations by [1], the QA manual for OpenVM 
project will be designed to visualise dependencies between individual outputs in the project creating a 
systematic process map of individual activities and their interrelationships with other activities in order 
to ensure the alignment of quality standards, processes and instruments. In order to gain the valuable 
outside perspective, the QA manual will also define a more efficient way of external quality 
assessment by voluntary experts.. 
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